Credible practice of modeling and simulation in healthcare: ten rules from a multidisciplinary perspective.

The complexities of modern biomedicine are rapidly increasing. Thus, modeling and simulation have become increasingly important as a strategy to understand and predict the trajectory of pathophysiology, disease genesis, and disease spread in support of clinical and policy decisions. In such cases, inappropriate or ill-placed trust in the model and simulation outcomes may result in negative outcomes, and hence illustrate the need to formalize the execution and communication of modeling and simulation practices. Although verification and validation have been generally accepted as significant components of a model's credibility, they cannot be assumed to equate to a holistic credible practice, which includes activities that can impact comprehension and in-depth examination inherent in the development and reuse of the models. For the past several years, the Committee on Credible Practice of Modeling and Simulation in Healthcare, an interdisciplinary group seeded from a U.S. interagency initiative, has worked to codify best practices. Here, we provide Ten Rules for credible practice of modeling and simulation in healthcare developed from a comparative analysis by the Committee's multidisciplinary membership, followed by a large stakeholder community survey. These rules establish a unified conceptual framework for modeling and simulation design, implementation, evaluation, dissemination and usage across the modeling and simulation life-cycle. While biomedical science and clinical care domains have somewhat different requirements and expectations for credible practice, our study converged on rules that would be useful across a broad swath of model types. In brief, the rules are: (1) Define context clearly. (2) Use contextually appropriate data. (3) Evaluate within context. (4) List limitations explicitly. (5) Use version control. (6) Document appropriately. (7) Disseminate broadly. (8) Get independent reviews. (9) Test competing implementations. (10) Conform to standards. Although some of these are common sense guidelines, we have found that many are often missed or misconstrued, even by seasoned practitioners. Computational models are already widely used in basic science to generate new biomedical knowledge. As they penetrate clinical care and healthcare policy, contributing to personalized and precision medicine, clinical safety will require established guidelines for the credible practice of modeling and simulation in healthcare.

Deciphering the "Art" in Modeling and Simulation of the Knee Joint: Overall Strategy.

Recent explorations of knee biomechanics have benefited from computational modeling, specifically leveraging advancements in finite element analysis and rigid body dynamics of joint and tissue mechanics. A large number of models have emerged with different levels of fidelity in anatomical and mechanical representation. Adapted modeling and simulation processes vary widely, based on justifiable choices in relation to anticipated use of the model. However, there are situations where modelers' decisions seem to be subjective, arbitrary, and difficult to rationalize. Regardless of the basis, these decisions form the "art" of modeling, which impact the conclusions of simulation-based studies on knee function. These decisions may also hinder the reproducibility of models and simulations, impeding their broader use in areas such as clinical decision making and personalized medicine. This document summarizes an ongoing project that aims to capture the modeling and simulation workflow in its entirety-operation procedures, deviations, models, by-products of modeling, simulation results, and comparative evaluations of case studies and applications. The ultimate goal of the project is to delineate the art of a cohort of knee modeling teams through a publicly accessible, transparent approach and begin to unravel the complex array of factors that may lead to a lack of reproducibility. This manuscript outlines our approach along with progress made so far. Potential implications on reproducibility, on science, engineering, and training of modeling and simulation, on modeling standards, and on regulatory affairs are also noted.

Credibility Evidence for Computational Patient Models Used in the Development of Physiological Closed-Loop Controlled Devices for Critical Care Medicine.

Physiological closed-loop controlled medical devices automatically adjust therapy delivered to a patient to adjust a measured physiological variable. In critical care scenarios, these types of devices could automate, for example, fluid resuscitation, drug delivery, mechanical ventilation, and/or anesthesia and sedation. Evidence from simulations using computational models of physiological systems can play a crucial role in the development of physiological closed-loop controlled devices; but the utility of this evidence will depend on the credibility of the computational model used. Computational models of physiological systems can be complex with numerous non-linearities, time-varying properties, and unknown parameters, which leads to challenges in model assessment. Given the wide range of potential uses of computational patient models in the design and evaluation of physiological closed-loop controlled systems, and the varying risks associated with the diverse uses, the specific model as well as the necessary evidence to make a model credible for a use case may vary. In this review, we examine the various uses of computational patient models in the design and evaluation of critical care physiological closed-loop controlled systems (e.g., hemodynamic stability, mechanical ventilation, anesthetic delivery) as well as the types of evidence (e.g., verification, validation, and uncertainty quantification activities) presented to support the model for that use. We then examine and discuss how a credibility assessment framework (American Society of Mechanical Engineers Verification and Validation Subcommittee, V&V 40 Verification and Validation in Computational Modeling of Medical Devices) for medical devices can be applied to computational patient models used to test physiological closed-loop controlled systems.

Assessing Computational Model Credibility Using a Risk-Based Framework: Application to Hemolysis in Centrifugal Blood Pumps.

Medical device manufacturers using computational modeling to support their device designs have traditionally been guided by internally developed modeling best practices. A lack of consensus on the evidentiary bar for model validation has hindered broader acceptance, particularly in regulatory areas. This has motivated the US Food and Drug Administration and the American Society of Mechanical Engineers (ASME), in partnership with medical device companies and software providers, to develop a structured approach for establishing the credibility of computational models for a specific use. Charged with this mission, the ASME V&V 40 Subcommittee on Verification and Validation (V&V) in Computational Modeling of Medical Devices developed a risk-informed credibility assessment framework; the main tenet of the framework is that the credibility requirements of a computational model should be commensurate with the risk associated with model use. This article provides an overview of the ASME V&V 40 standard and an example of the framework applied to a generic centrifugal blood pump, emphasizing how experimental evidence from in vitro testing can support computational modeling for device evaluation. Two different contexts of use for the same model are presented, which illustrate how model risk impacts the requirements on the V&V activities and outcomes.

Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories.

Protecting and promoting public health is the mission of the U.S. Food and Drug Administration (FDA). FDA's Center for Devices and Radiological Health (CDRH), which regulates medical devices marketed in the U.S., envisions itself as the world's leader in medical device innovation and regulatory science-the development of new methods, standards, and approaches to assess the safety, efficacy, quality, and performance of medical devices. Traditionally, bench testing, animal studies, and clinical trials have been the main sources of evidence for getting medical devices on the market in the U.S. In recent years, however, computational modeling has become an increasingly powerful tool for evaluating medical devices, complementing bench, animal and clinical methods. Moreover, computational modeling methods are increasingly being used within software platforms, serving as clinical decision support tools, and are being embedded in medical devices. Because of its reach and huge potential, computational modeling has been identified as a priority by CDRH, and indeed by FDA's leadership. Therefore, the Office of Science and Engineering Laboratories (OSEL)-the research arm of CDRH-has committed significant resources to transforming computational modeling from a valuable scientific tool to a valuable regulatory tool, and developing mechanisms to rely more on digital evidence in place of other evidence. This article introduces the role of computational modeling for medical devices, describes OSEL's ongoing research, and overviews how evidence from computational modeling (i.e., digital evidence) has been used in regulatory submissions by industry to CDRH in recent years. It concludes by discussing the potential future role for computational modeling and digital evidence in medical devices.

Applying a Computational Model Credibility Framework to Physiological Closed-loop Controlled Medical Device Testing.

Physiological closed-loop controlled medical devices are safety-critical systems that combine patient monitors with therapy delivery devices to automatically titrate therapy to meet a patient's current need. Computational models of physiological systems can be used to test these devices and generate pre-clinical evidence of safety and performance before using the devices on patients. The credibility, utility, and acceptability of such model-based test results will depend on, among other factors, the computational model used. We examine how a recently developed risk-informed framework for establishing the credibility of computational models in medical device applications can be applied in the evaluation of physiological closed-loop controlled devices.

Use of the FDA nozzle model to illustrate validation techniques in computational fluid dynamics (CFD) simulations.

A "credible" computational fluid dynamics (CFD) model has the potential to provide a meaningful evaluation of safety in medical devices. One major challenge in establishing "model credibility" is to determine the required degree of similarity between the model and experimental results for the model to be considered sufficiently validated. This study proposes a "threshold-based" validation approach that provides a well-defined acceptance criteria, which is a function of how close the simulation and experimental results are to the safety threshold, for establishing the model validity. The validation criteria developed following the threshold approach is not only a function of Comparison Error, E (which is the difference between experiments and simulations) but also takes in to account the risk to patient safety because of E. The method is applicable for scenarios in which a safety threshold can be clearly defined (e.g., the viscous shear-stress threshold for hemolysis in blood contacting devices). The applicability of the new validation approach was tested on the FDA nozzle geometry. The context of use (COU) was to evaluate if the instantaneous viscous shear stress in the nozzle geometry at Reynolds numbers (Re) of 3500 and 6500 was below the commonly accepted threshold for hemolysis. The CFD results ("S") of velocity and viscous shear stress were compared with inter-laboratory experimental measurements ("D"). The uncertainties in the CFD and experimental results due to input parameter uncertainties were quantified following the ASME V&V 20 standard. The CFD models for both Re = 3500 and 6500 could not be sufficiently validated by performing a direct comparison between CFD and experimental results using the Student's t-test. However, following the threshold-based approach, a Student's t-test comparing |S-D| and |Threshold-S| showed that relative to the threshold, the CFD and experimental datasets for Re = 3500 were statistically similar and the model could be considered sufficiently validated for the COU. However, for Re = 6500, at certain locations where the shear stress is close the hemolysis threshold, the CFD model could not be considered sufficiently validated for the COU. Our analysis showed that the model could be sufficiently validated either by reducing the uncertainties in experiments, simulations, and the threshold or by increasing the sample size for the experiments and simulations. The threshold approach can be applied to all types of computational models and provides an objective way of determining model credibility and for evaluating medical devices.

The Role of Computational Modeling and Simulation in the Total Product Life Cycle of Peripheral Vascular Devices.

The total product life cycle (TPLC) of medical devices has been defined by four stages: discovery and ideation, regulatory decision, product launch, and postmarket monitoring. Manufacturers of medical devices intended for use in the peripheral vasculature, such as stents, inferior vena cava (IVC) filters, and stent-grafts, mainly use computational modeling and simulation (CM&S) to aid device development and design optimization, supplement bench testing for regulatory decisions, and assess postmarket changes or failures. For example, computational solid mechanics and fluid dynamics enable the investigation of design limitations in the ideation stage. To supplement bench data in regulatory submissions, manufactures can evaluate the effects of anatomical characteristics and expected in vivo loading environment on device performance. Manufacturers might also harness CM&S to aid root-cause analyses that are necessary when failures occur postmarket, when the device is exposed to broad clinical use. Once identified, CM&S tools can then be used for redesign to address the failure mode and re-establish the performance profile with the appropriate models. The Center for Devices and Radiological Health (CDRH) wants to advance the use of CM&S for medical devices and supports the development of virtual physiological patients, clinical trial simulations, and personalized medicine. Thus, the purpose of this paper is to describe specific examples of how CM&S is currently used to support regulatory submissions at different phases of the TPLC and to present some of the stakeholder-led initiatives for advancing CM&S for regulatory decision-making.

Design considerations for studies of the biomechanical environment of the femoropopliteal arteries.

OBJECTIVE: The purpose of this study was to review the available literature regarding the biomechanics of the superficial femoral artery (SFA) and popliteal artery (PA) in patients with peripheral arterial disease (PAD). Stents are one of many available therapies used to treat patients with PAD. Because stents are permanent implants, they undergo a variety of deformations as patients go about their daily activities such as walking, sitting in a chair, or climbing stairs. As a part of the marketing application for United States Food and Drug Administration approval, stents need to be evaluated for long-term durability under a variety of loading modes. The information available in the literature provides direction for such evaluation. METHODS: We performed a literature search of the PubMed database looking for "key vessel" and "mechanics" (all fields) or "deformation" (all fields) or "flexion" (all fields) or "mechanical environment" (all fields) or "tortuosity" (all fields) or "dynamics" (all fields) or "forces" (all fields), where the "key vessel" was "Femoral Artery," "Superficial Femoral Artery," "Popliteal Artery," and "Femoropopliteal." RESULTS: Using a decision tree, we found 12 relevant articles that focused solely on the nonradial cyclic deformations associated with musculoskeletal motion. Despite the many limitations associated with combining these studies, we learned that under walking conditions, the proximal and mid-SFA deforms, on average, by shortening in the axial direction 4.0%, by twisting 2.1°/cm, and by bending 72.1 mm; the distal SFA and proximal PA deform by shortening in the axial direction 13.9%, by twisting 3.5°/cm, and by being pinched such that the aspect ratio of the lumen changes 4.6%. The distal PA deforms by shortening in the axial direction 12.3%, by twisting 3.5°/cm, by bending 22.1 mm, and by being pinched such that the aspect ratio of the lumen changes 12.5%. CONCLUSIONS: A review of the current literature reveals heterogeneous study designs that confound interpretation. Studies included different physiologic settings from young to mature participants, participants with and without disease, and cadavers. Investigators used a range of imaging modalities and definitions of arterial segments, which affected our ability to compile the data as we learned that deformations vary according to the specific anatomic location within the SFA/PA. As a result of this analysis, we identified design considerations for future studies, because although this work has been valuable and significant, there are many limitations with the currently available data such that all we know about the SFA/PA environment is that we don't know.

Considerations for reporting finite element analysis studies in biomechanics.

Simulation-based medicine and the development of complex computer models of biological structures is becoming ubiquitous for advancing biomedical engineering and clinical research. Finite element analysis (FEA) has been widely used in the last few decades to understand and predict biomechanical phenomena. Modeling and simulation approaches in biomechanics are highly interdisciplinary, involving novice and skilled developers in all areas of biomedical engineering and biology. While recent advances in model development and simulation platforms offer a wide range of tools to investigators, the decision making process during modeling and simulation has become more opaque. Hence, reliability of such models used for medical decision making and for driving multiscale analysis comes into question. Establishing guidelines for model development and dissemination is a daunting task, particularly with the complex and convoluted models used in FEA. Nonetheless, if better reporting can be established, researchers will have a better understanding of a model's value and the potential for reusability through sharing will be bolstered. Thus, the goal of this document is to identify resources and considerate reporting parameters for FEA studies in biomechanics. These entail various levels of reporting parameters for model identification, model structure, simulation structure, verification, validation, and availability. While we recognize that it may not be possible to provide and detail all of the reporting considerations presented, it is possible to establish a level of confidence with selective use of these parameters. More detailed reporting, however, can establish an explicit outline of the decision-making process in simulation-based analysis for enhanced reproducibility, reusability, and sharing.

The influence of gender and aortic aneurysm size on eligibility for endovascular abdominal aortic aneurysm repair.

OBJECTIVES: The purpose of this study was to compare the eligibility of men and women with infrarenal abdominal aortic aneurysms (AAAs) for on-label endovascular aneurysm repair (EVAR) as part of the clinician-Food & Drug Administration (FDA) collaborative effort, the Characterization of Human Aortic Anatomy Project (CHAP). METHODS: Computed tomography (CT) scans with 3D reconstruction from a single institution obtained between July 1996 and December 2009, including standardized measurements by a blinded third-party (M2S, West Lebanon, NH) were examined. For inclusion, abdominal aortic aneurysm (AAA) had to be infrarenal, unrepaired, and >5 cm, or 4 cm to 5 cm if the orthogonal sac diameter was more than twice the aortic diameter at the renal level. Scans were included regardless of subsequent EVAR, open repair, or lack of treatment. One thousand sixty-three unique, unrepaired AAAs were analyzed. RESULTS: Neck length, diameter, and angulation differ for women (P < .001) even after adjustment for patient age and AAA size. EVAR eligibility based on device Instructions for Use (IFU) criterion is affected by gender. Neck length <15 mm was found in 47% of men and 63% of women. Neck angulation exceeding 60 degrees was found in 12% of men and 26% of women. Minimum iliac diameter of 6 mm was found in 35% of men and 55% of women. Only 32% of men and 12% of women met all three neck criterion and had iliac lumen diameters >6 mm. Logistic regression modeling shows that older patient age (odds ratio [OR], 0.84 per decade), increased aneurysm diameter (OR, 0.70 per cm), and female gender (OR, 0.4) are each independently associated with decreased odds of meeting all device IFU neck criterion (P < .05). EVAR eligibility by neck criterion does not decline significantly until AAA size exceeds 5.5 cm in women and 6.5 cm in men. CONCLUSION: Women are significantly less likely to meet device IFU criterion for EVAR. Aortic neck criteria and iliac access are important for men and women, but more women than men fail to meet IFU criterion. Devices that accommodate shorter infrarenal AAA neck length will have the greatest impact on expanding on-label EVAR regardless of gender. Lower profile devices and those that accommodate higher neck angulation are expected to expand EVAR eligibility further for women. EVAR eligibility is unlikely to be lost as AAAs enlarge to 5.5 cm in women and 6.5 cm in men. Observation of small AAAs until they reach the standard threshold size for repair should not compromise EVAR eligibility.

Circumferential and longitudinal cyclic strain of the human thoracic aorta: age-related changes.

OBJECTIVE: We developed a novel method using anatomic markers along the thoracic aorta to accurately quantify longitudinal and circumferential cyclic strain in nondiseased thoracic aortas during the cardiac cycle and to compute age-related changes of the human thoracic aorta. METHODS: Changes in thoracic aorta cyclic strains were quantified using cardiac-gated computed tomography image data of 14 patients (aged 35 to 80 years) with no visible aortic pathology (aneurysms or dissection). We measured the diameter and circumferential cyclic strain in the arch and descending thoracic aorta (DTA), the longitudinal cyclic strain along the DTA, and changes in arch length and motion of the ascending aorta relative to the DTA. Diameters were computed distal to the left coronary artery, proximal and distal to the brachiocephalic trunk, and distal to the left common carotid, left subclavian, and the first and seventh intercostal arteries. Cyclic strains were computed using the Green-Lagrange strain tensor. Arch length was defined along the vessel centerline from the left coronary artery to the first intercostal artery. The length of the DTA was defined along the vessel centerline from the first to seventh intercostal artery. Longitudinal cyclic strain was quantified as the difference between the systolic and diastolic DTA lengths divided by the diastolic DTA length. Comparisons were made between seven younger (age, 41 +/- 7 years; 5 men) and seven older (age, 68 +/- 6 years; 5 men) patients. RESULTS: The average increase of diameters of the thoracic aorta was 14% with age from the younger to the older (mean age, 41 vs 68 years) group. The average circumferential cyclic strain of the thoracic aorta decreased by 55% with age from the younger to the older group. The longitudinal cyclic strain decreased with age by 50% from the younger to older group (2.0% +/- 0.4% vs 1.0% +/- 1%, P = .03). The arch length increased by 14% with age from the younger to the older group (134 +/- 17 mm vs 152 +/- 10 mm, P = .03). CONCLUSIONS: The thoracic aorta enlarges circumferentially and axially and deforms significantly less in the circumferential and longitudinal directions with increasing age. To our knowledge, this is the first quantitative description of in vivo longitudinal cyclic strain and length changes for the human thoracic aorta, creating a foundation for standards in reporting data related to in vivo deformation and may have significant implications in endoaortic device design, testing, and stability.